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ABSTRACT 

A   test   based   on   the   Operations  Analysis    Curriculum 
at   the  United   States    Naval   Postgraduate   School   was 
administered  to   104    Naval   Officers.      All   examinees   were 
graduates   or   students   of   the  Operations  Analysis    Curricu- 
lum  and/or   officers  holding   Operations  Anslysts  billets   in 
the   Navy.      The   sub-sample,    34    examinees,    consisting   of 
officers   holding   Operations  Analysts   billets   and/or   Operations 
Analysis   graduates   was   not   sufficient   to    make  adequate 
statistical   determination   of   the  measure   of   effectiveness 
proposed  in  a   suggested   methodology.      The   data   gathered 
did   crudely   support  hypothesized  learning  and  forgetting 
curves   and   suggested   that   the   effectiveness   of   Operations 
Analysis   graduates  assigned  directly  to   Operations  Analysts 
billets   immediately  after   graduation  is   much   enhanced   compared 
to   graduates   who   are   returned  first   to   fleet   operational 
billets.      The   effectiveness   of   Operations  Analysis   trained 
officers   in   Operational  Analyst  billets    was   shown   to   be 
quantitatively  and   subjectively  significantly   superior  to   those 
with   no   formal   Operations  Analysis   training.       These   results 
indicate  that    Naval  assignment   policies    should  be   reviewed 
in  hopes   of  assigning   more   Operations  Analysis   trained 
officers    (consistent   with  other   requirements)    to   these 
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billets.      Further   investigation  of   the  results    of   the  test 
vehicle  and   other   statistics   common  to    Operations  Analysis 
graduates   yielded  a  feasible  procedure   with   which  to 
augment   the   screening   of   prospective   Operations  Analysis 
students.      Final   Quality   Point   Rating,    an  acceptable 
measure  of   performance,    had  a    ,614   correlation  with   four 
readily  available   statistics. 

This    evaluation  suggests   that   further   study  in  this 
area   has   great   promise  in   yielding  useful   measures   of 
effectiveness   for   all   personnel   filling  billets   requiring   post- 
graduate  education,    provided  a   more   effective   method   is 
employed  to   insure   completion  of   the  required  test 
ins tr um  ent  ( s ) . 
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CHAPTER    I 
INTRODUCTION 

In  May   of   196i+   Commander    P.    D.    Roman,    and 
Lieutenant   Commanders    K.    D.    Russell  and   J,    M.    Dunlop 
posed  a  highly  interesting  and  promising    methodology   designed 
to    evaluate  the   effectiveness   of   the  utilization  of   technically 
trained  personnel   within  the    Navy.j_llJ      The   personnel 
involved  in  this    suggested  Methodology   were   Naval   Officers 
who   have   received   postgraduate   education.      The   main 
motivation  for   this   proposed   model   was   the  fact   that   no 
previous   quantitative   study  had  been   made  to   determine   if   a 
better   balance   of    career   duty  assignments    should  be   devised 
which   would  allow   officers   to   attain  the   operational   knowledge 
and   experience   requisite  to   Military   Command  and   simultane- 
ously approach    maximum    effective  use   of   their   technical   skills 
(sub-specialties).      Because  the  acquisition  of   technical   skills 
is   costly,    as    well   as   necessary,    finding  an   optimal   procedure 
to   utilize  these   skills,    consistent   with   other   requisites,    is 
virtually  a    "must."      The   pure   rationality  of    cost-effectiveness 
alone   supports   this   supposition. 

Commander   Roman,    et„    al.,    constructed  a   test   instru- 
ment   modeled   on  the   Operations  Analysis   curriculum   at    USNPGS 
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which   would,    hopefully,    measure   the  native  ability  and 
technical   knowledge   of    Naval   Officers   who   are  holding 
Operations  Analysis   billets   and/or   graduates   of   the  Oper- 
ations Analysis   Curriculum.      It   was   their  hypothesis    that 
by   categorizing   these   officers   into   several   distinct   groups 
the   statistical   results   of   their   performance   on  this   test 
would   lead  to   a    more   efficient   system    of   career   planning 
for   these   officers.      If   this   procedure  proved   successful 
for   the   Operations  Analysis    sub-specialty,    it   would  obviously 
have  applications   for  all   officers   who   are  technically   trained. 
The  authors    of   the  referenced    methodology  have  presented 
an  admirable   treatise  on  the  need  for   the   execution   of    such 
a   quantitative   study. 

In   September   of   196i+   the  authors   of   this   paper   mimeo- 
graphed the  test   instrument  and   distributed  it  to   officers 
holding  Operations  Analysis    "PM   coded  billets   and/or   graduates 
of   the   Operations  Analysis    Curriculum   at   USNPGS.      A 
total   of  34   returns   from   a   population  of   197   officers   was 
realized.      In  January   of    1965   the   test   was   also   administered 
to   36    second  year   Operations  Analysis    students   at   USNPGS 
scheduled   to   graduate  in  May  1965.      In  November   of   I964, 
34   first   year    Operations  Analysis    students    completed  the 
aptitude  portion  of   the  test   vehicle. 
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The  purpose  of   this   paper   is   three-folds 

I .  To   determine   the  adequacy   of   the  test   as   a 
measuring  instrument,    i.e.,    does   it   measure   what   it   is 
supposed  to    measure? 

II.  To    carry  out,    where  possible,    the   statistical 
procedures    suggested  by   Commander   Roman,    et.    al.,    and 
analyse  their   usefulness   and/or   implications. 

III.  To   analyse  the   results   of   a   multiple  linear 
regression  analysis   performed  on   the   CDC   I60i+    Computer 
to   develop  a   statistical    means   to   augment   selection 
procedures   for   input   to   the  Operations  Anslysis   Curriculum 

The  following   chapter   details   the  analysis    of   the   data 
for   each  of  these  purposes. 


CHAPTER    II 
THE   ANALYSIS 

I.       Determination   of   the  adequacy   of   the  test   instrument. 

The   test   instrument   used  in   this   study   consists   of   a 
background   questionaire  and   three   test   parts.      The   question- 
aire   is   designed   to    obtain  the   examinee's    educational  background, 
a  history   of  his    duty  assignments,    a   listing   of   graduate  and 
undergraduate   courses    completed  and   the   examinees   opinions' 
as   to    what   courses  he  is   lacking   that   are   required  in  his 
billet.      The   main  purpose  of   this   part   of   the  test   is   to 
stratify   the   examinee  according   to    educational   background. 
The   other   information  gained   will   provide   further   refinement 
of   this    stratification.      The  adequacy   of   this   questionaire  is 
purely   subjective.       The  authors   feel   that   the  information 
provided  by   the   questionaire   will  allow   assignment   of   testees 
to   logical   categories   of   interest   to   this    study. 

Part   I    of   this   test   is   an  aptitude   test   designed   to 
provide  a   measure   of   the   examinee's   ability  in  problem    solving, 
the  only    major   factor  accepted  as    effecting   current   measures 
of   intelligence.         This   test   was   constructed,    from   investigations 

Editorial.      Federal   Education,    You're  in  the   Classroom 
Now.      Time,    83,    3,    January  17,    1964 :     72 
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of   aptitudes    of   high   level   personnel,    under   the   direction   of 
Dr.    J.    P.    Guilford  at   the   University   of   Southern   California, 
Los  Angeles.  |1,6,9|       The  authors    will  assume   that   the 
results   of   this    exhaustive  and  authoritative   study  has   led  to 
the  formulation  of   an   excellent   test   to    measure  native  ability. 
It   follows   that   the   results    of   this   test   will  allow    examinees 
to   be   rated  according  to   inherent   abilities.      This   stratification, 
coupled   with  background  areas,    will   provide  an   excellent    means 
for    comparison  of   technical   skills   in  the   selected   categories. 

Part   II    of   the  test   deals   strictly   with   the  retention 
of   fundamental   concepts   of    the  basic   courses   within  the 
Operations  Analysis   Curriculum:      namely,    Advanced   Calculus, 
Linear  Algebra   and   Probability   theory.      This   test   was 
formulated  from   suggestions   of   Professors   of   the  Mathematics, 
Physics   and   Operations  Analysis    Departments   of   USNPGS. 
It   is   indeed  a   moot   question  as   to   whether   or   not  a   good 
working  knowledge  of   these   courses   measures,    with  any   degree 
of  accuracy,    the  ability   or   success   of   an  individual  as   an 
Operations  Analyst.      There   can  be  little  doubt,    however, 
that   proficiency   in  these  fields   does   reflect   some   measure  of 
the   examinees   technical  abilities.      Beacuse   we  are  once  again 
caught  in  a   subjective    (or   qualitative)    area,    at   this   point   we 
will  assume  that   the   combined   opinions    of   these   recognized 
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educators   provides   a   good   cross   section  of   the  technical 
knowledge   required   of   an   effective   Operations  Analyst,    and 
the  results   of   Part   II    of   this   test   -will  yield,    at   least,    a 
relative   measure  among  the  groups   of   officers   of   their 
abilities   as    Operations  Analysts.      Relative  performance  is, 
after   all,    a   very  important   aspect   in   our   real   world  and 
this   relative  performance  is   in   essence   one   major   objective 
of   this   study. 

Part   III    is   a  practical  test   designed  to    evaluate   the 
examinees'     ability   to   recognize  the  applicability   of  a   class    of 
methods   to    specific   problems.      The   situations   presented  in 
these  problems    do   not  have   clean-cut   answers,    but   are 
designed  to   determine  how   familiar   an   examinee  is   with  an 
"accepted   or   proven"   Methodology  as   related   to    well-known 
Operations  Analysis   problems.      The  answers    to   the   situations 
posed  are  nothing   more  than  a   mean   combination  of   opinions 
expressed  by  noted  analysts   and  professors  at   USNPGS. 
Once  again   we  are  using  the  general   reasoning   of   the 
preceding  paragraph  in   stating  that   this   portion  of   the  test 
will   display  a   relative   measure  of   how   familiar   an   examinee 
is   with   the    "accepted"   tactics   of   Operations  Analysis. 

Thus   far   we  have  been   concerned   with   the  question   of 
whether   the  test   will    measure   what   we   want   it   to    measure. 

6 


Is   it   a   valid  indication  of    the  technical  abilities    we  are 
trying   to    measure?      To    this   point   we   have   tried   to   answer 
these   questions   in   the  affirmative  by   the   use  of    subjective 
reasoning.      This   procedure   is   necessary  because  the  deter- 
mination  of   test   validity   does   not   readily   or   easily   lend 

2 
itself   to    meaningful   quantitative  analysis.         Despite  this 

dilemma,    certain    mathematical   techniques   do   allow   a   degree 

of    quantitative   determination  of   adequacy  to   be   calculated. 

The   entire  question  of   determining   the  adequacy  of   this   test 

could  be   measured  by   two   broad   criteria!    VALIDITY   and 

RELIABILITY.      Reliability   is   defined  as   a   measure   of   how 

faithfully   the  test   allows   the   examinee  to   display   the  true 

percentage   of   the   questions   presented  in  the   test   to   which 

he  actually  knows    the  answers.      A   mathematical   presentation 

of   this   reliability   follows. 

Doctor   J,    Pc    Guilford  has   developed  a   mathematical 

model   to    evaluate  the  reliability   of   any   test.      Because  the 

entire   development   is    extremely  lengthy   we   shall   present 

only  the  basic   assumptions   and  final  forms   of   the   reliability 

3 
equations  • 

2 
Guilford,    «J.    P.    Psychometric   Methods,    McGraw   Hill 

Co.,    New     York,  1954s      36 
3Ibid:      344-409 


The   effects    of   these  variations   have  been   reduced  as 


much  as   possible  by  the  use  of  an  a   priori   weighting   of   raw 
total   scores. 


(1) 


where: 


,„R(K-1)    -   W 
K 

S   =  Adjusted   Score, 
R   =    Number   of   Right  Answers, 
W  =   Number   of  Wrong  Answers, 
K  =    Number   of  Alternate  Responses   to    each 
item. 


This   method   of    scoring  is   designed  to   nullify  the   small   but 
not   necessarily   minute  possibility  that   the   examinee   may 
guess    the   correct  answer. 

Doctor   Guilford  has   postulated   that,    in  theory,    a 
regression   equation   could  be   calculated  between   observed 
scores   and  the  true   score    ( exactly  how   many   questions   the 
student   knows     the  answers   to)    as   depicted  in  Figure   1. 
In  the  development   of   this   theory   Dr.    Guilford  postulated 
that  in  any  academic   area  a   test   of   infinite  length    would  be 
necessary   to   completely  describe  an   examinee's  knowledge  of 
the   subject.      In  like   manner   the  authors   have   denoted   true 
score  by   the   symbol     QQ  . 


>  in 


ac 


Uj 
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Regression  line. 

Standard   deviation  of   the 
error  in  the   measurement, 


True   Score  =  QO 


FIGURE    1 
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The  following  assumptions    were  used  in  the   mathematical 
development: 

I        Me       =  Mean   error   =   0. 

H     K»e      -   Correlation  of   true  and   error   scores   =   0. 

III  \et  £»«    -   Error   score   correlation  in  any  two   forms 

of   the   same   test   =   0. 

IV  Distribution  of    errors   is   normal. 
It   then  follows   that: 

Mqq  =   M^  +   Me  =   M^. 

and 

Moo    =  M^     +  ^Je  where  \|  £      =   variance   of   6, 

This   leads   to   the  logical   definition  of   reliability   of   a   test 

as   the  proportion  of   true  variance  in   obtained   test   scores 

or         (2)  fo    -    ^; 

where:  (  h  s    Test   reliability,  Q-fef  -  lj 

V©o  =■  True   variance   of   test   scores 
sfit  =   Observed   variance   of   test    scores. 

Because   of   the   difficulty  in   obtaining   \J  qq   equation    (2)    has 

been   reduced    (closely  approximated)    as   follows: 

o)  \u  -  l  -  -5£j 

where:  M  =    Difference  between  the   scores   on   even 

n         items   and   odd   items   on  the   test, 

J  =   Variance   of   these   differences, 

^JZ  =  Variance   of   obtained  total   scores. 

The   variables   in   equation    (3)    can  be   readily   calculated  for 

any  test. 
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A  final  consideration   must  be  made  in  view   of  the 
fact   that   the  test  in  question  was   timed  and  therefore 
speeded  to   some  extent.      Speed  does   detract  from 
performance  which  we  do  not  want  reflected  in  reliability. 
The  final  form    of   this   reliability   equation  is 

(4)  Fn\      =     Yt-b    "    ^="*    ) 

where:  i/y\     -  Reliability   of   slightly  speeded  test, 

Y-kt   —  sls  before, 

UL     =   mean  number   of  unattempted  items, 

rr-^ 

\|^g  =   variance  of   total   test   error   scores. 

Equation    (i+)    is   a   close  approximation  to   true   reliability,    as 

previously   defined,    of  a   speeded  test   provided  that: 

\J^    ^.0*3  where:    \|  ^     =   Standard  deviation 

_ir~^  of  unattempted  test 

^t£  items, 

and 

« —  VW     —  Standard   deviation  of 

JUL       y  i    -y                                                            test  items  answered 

^j —        —  '*  ^J                                                            incorrectly. 

As   suggested  by   Commander   Roman,    et.    al.,    examinees 
were  placed  in  the  following   categories: 

(1)    USNPGS    Students   nearing   completion  of   their 
final   year   of   the   curriculum. 

(1A)    USNPGS   Students   in   second   term    of  first   year 
of   Operations  Analysis   Curriculum. 
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(2A)    Graduates   who   have  been  assigned    (and  are  in) 
directly  to   Operations  Analysis  billets* 

(2B)    Graduates   who  have  never  been  associated  with 
Operations  Analysis  billets,, 

(2C)    Graduates   who   have   completed  a   direct   assignment 
in   Operations  Analysis  billets   and  are   now   in  unassociated 
activities, 

(2D)    Graduates   who   were  not   immediately  assigned 
Operations  Analyst  billets   but   are  now   serving  in  that 
capacity. 

(3)    Non-Graduates    who  are  presently   serving  in 
Operations  Analyst  billets. 

Clearly   all    examinees   involved  in  this   study  fall  into   one 
and   only   one   of   these   categories.      The   chosen    method   of 
attack  is   to    compare  various   categories  by   means   of   forgetting, 
learning  and  re-learning   curves   in  hopes   of   deducing  a   useful 
measure  of    effectiveness.      As  a   common  basic   for   comparison, 
all   examinees   were   stratified  by   comparing  their   results   of 
Part   I    (Inherent  Abilities)    with  the   mean   score   of   Part   I 
for    Category  £«      This   procedure  is   logical  because  present 
students    should  be  more  familiar   with   the  basic   fundamentals 
of   the   courses   test   Parts   II    and  III    are   concerned   with. 
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To  be   specific,    a  ratio   of  an  individual's   score  on  Part  I 
to   the   mean   score   of   Category  I    on   Part   I    was   multiplied 
by   the   scores   on  Part   II    and  III    and   this   figure   compared 
to   the  same   mean  score  attained  on  Parts   II   and  III   by 
Category  I . 

From    equations    (3)    and    (4)    and   the  information  displayed 
in  Appendix   II    the   reliability   of   this    entire   test    was   calcul- 
ated  to   be  /C    =    .9024.      Category  I    students   were   chosen  as 
a   reliability  base   for   the  following  reasons  $ 

1.  All   data   and   comparisons   are   to   be  based  on  their 
performance. 

2.  Strict   time  limitations   were  imposed   on  these 
testees   and   we  feel  that   their   test   conditions   are   more  in 

keeping  with   the   stipulations   and  assumptions    made  in  the 

4 
development   of   the  reliability   equations. 

In  the  final  analysis,    validity  has   to   do    with   what   test 

scores    measure  and   what   they   will  predict.      A   score   is   valid 

for   predicting  anything   with   which  it   correlates,    where 

"anything"    does   not   include   the   score  itself,    for  a   self 

5 
prediction  has   to   do   with   reliability.         We   chose  to   predict 

Guilford,    «J0    P.    Psychometric   Methods,    McGraw    Hill 
Co.,    Inc.,    New   York,    1954:      366 

5Ibid:      398 
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final   graduate  level   QUALITY    POINT   RATING*   of 
Operations  Analysis   graduates   of    USNPGS.      Here   we 
have  assumed  that   the  degree  of   successful  completion  of 
the  curriculum    (hence  a   measure   of   technical   ability)    is 
measured  by  the  reliability  of   the   criterion  of    QoP.R.; 
although   the  reliability   of    Q.P.R.    as   a   criterion  is   not 
known,    it  is   the  only   quantitative   measure   we  have  for 
comparison  and  have  therefore  assumed,    for   the  purposed 
of   this   study,    it   to   be  1.0.      The  results   of   a   multiple 
linear   regression  analysis    (Appendix  IV,     pp<,50  )    shows   that 
the   correlation   coefficient   of    Q0P.R0    to   various   parts    of 
the  test   scores   is  a   minimum   of    .5978   and  a   maximum   of 
.6li+2.      As    shown  in  Appendix  IV,    it   was   found  that   Part 
II   versus   Time  in  Operations  Analyst   billet   or   Time   since 
Graduation,    Part   I   and   Category  type  had  a   multiple 
correlation  of    .8210,    indicating  that   technical  knowledge  is 
extremely  dependent  upon  inherent  ability  and  the  way  time 
is   utilized  after  aquisition  of   these  abilities.      Since  the 
reliability   of    this   test  is   a  high    .9022+   and  its   overall 
correlation  to   a   real   life   criterion  is   high,    it   follows   that 
the  quantitative  adequacy  of   this   test  is   very  high. 


*   Hereafter   referred  to  as    Q.P.R.,    with  a   maximum   of 
3.0. 
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Within  the   assumptions    stated,    the   quantitative 
reliability  and   validity   calculation   and   the   previous    subjective 
verifications,    the   overall   adequacy   of    this    test   is    excellent. 

One  final   word   on   this    "sticky"    subject.      This   test    was 
sent   to   Mr.    R.Pr    Richardson,    Head   Operations   Analyst   for 
LING-TEMPCO-VAUGHT   ASTRONAUTICS   in   Dallas,    Texas 
for  his    comment   as   well  as   the   reaction   of   his   fellow    employees. 
Mr.    Richardson  replied   that   in  his   opinion,    and   the   opinion   of 
his    collegues,    the   overall   design   of   the   test   should   yield  a 
practical   relative   measure   of   the  technical   tools    of    Operations 
Analysts . 

II.       Comments    on   the   suggested   method  and   proposed   measure 
of    ef f ecticeness. 

In  the   suggested   methodology,    two   families   of    curves   were 
to   be  plotted  and   studied   in  hopes    of   arriving  at   a   useful    meas- 
ure  of    effectiveness.      The   first   family   of    curves      was   to   be 
a   plot    of    weighted  adjusted   score    (as    described   in   I    above)    for 
categories   2A,    2B,    2C   versus   time.      The   courses    were   post- 
ulated  to   appear   as   in   Figure   2. 
d 

Q_  =   level   of   knowledge   for 
c>  a  _    A     category   1   students. 
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The  second  family  of   courses   were  to  be  concerned  with 
the  process   of    learning  and  relearning  basic   technological 
knowledge  and  were  hypothesized  to  appear  as  in  Figure  3. 
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FIGURE   3 
The  plots    of   Figure  2,  if   correct,    would  indicate  that 
intervening   duty  assignments   between  acquisition  of   technical 
knowledge  and  application  in  the  given  field   results   in  a 
larger   loss    of   the  tools   required.      A  useful  M.O.E.    was   to 
be   constructed  from   the  type  graphs   of   figure  3.      For 
example:      if    "b"   in  figure  3   indicates   the  ordinal   value   of 
the  asymptote   of    curve    E,    the  ratio   of   curriculum   time   to 
curriculum   time  plus   time   required  for   category  2D    (graduates 
who  are   essentially  re-learning  the  technical  tools)    would 
yield  a   value  between  0   and   1,    a  higher   value   correlating  to 
a   more   effective  utilization  of   the  technical   resorurce  and 
would  imply  a   certain  percentage   of   effectiveness.      If   this 
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ratio    were  to   be   drastically   low   it   would   demonstrate  the 
need  for   assigning   the   technically   trained   to   associated 
billets   immediately   after   graduation   to    more   effectively 
utilize   the   costly  process   of    educating  these  individuals. 

Appendix  III    contains   plots    of    weighted  adjusted 
scores    on   Part   II   versus    Time   for   the  above   mentioned 
categories.      These  graphs   have  the  approximate   shapes   of 
Figures   2   and  3  •      At   this   point   this   particular   plan  of 
attack  breaks   down.      As   previ  ously   stated,    34   returns 
out   of  a   population  of   197    was   realized.      This   resulted  in 
a   maximum   of   14   points   for   the   learning   curve  for    category 
3   and  a     minimum   of   two   points   for   category  2C.      It  is   a 
well   known  fact   that   little   credence   can  be  placed   in  a 
statistical   procedure   unless: 

I.  The   sample   size  is   very  large    (.9   to    .95)    in  proportion 
to   the  population  size,    or 

II.  One   can   safely  assume  knowledge   of   the   exact   distri- 
bution of   the   random   variables    observed,    and/or  have   control 

7 
over   the   method   of    selecting   the   sample   size. 

Burington,    R.    So,    and  May,    D.    C0      Handbook  of 
Probability  and   Statistics    with   Tables,    Handbook  Publishers, 
Inc.,    Sandusky,    Ohio   1953^      170 


7Ibid:      170-178 


16 


Clearly  none  of   these  proposed  curves   contain  data 
satisfying  any   of   these   conditions.      At    most,    one   can  say 
that   the  data  so  far  gathered  crudely  support   the  hypothe- 
sized curves   and  that  future  study  along  these  lines  has 
more   than  a   possibility  of   being  fruitful. 

If,    however,    we   consider   the   sample   size  to    consist 
of   only   two   groups,    Operations  Analysis   graduates   and   Non- 
Operations  Analysis   graduates,    several   meaningful  implications 
can  be  garnered  from   the  test   results.       (Group   I    Operations 
Analysis  Graduates,    Group  II   Non-Operations  Analysis 
Graduates)  .       The   mean   score  on   Part   I   for   Group   II    was 
6.0   points  below    Group   I   indicating  a   significantly  lower 
proficiency  in  the  area   of   problem    solving.      The   mean   scores 
on  Parts   II   and  III    were  respectively,  3  arid  2+  times  as  high 
for   Group   I    as   for   Group   II.      These   mean  scores   had  a 
standard  deviation  of  less  than  7.0   for  the   "worst"   case. 
Group   I's   comments   on  the   courses   that   they   thought   they 
were   lacking   were   of   a  highly  specialized  nature    (Dynamic 
Programming,    Specialized  Methods   of    Cost-Effectiveness, 
etc.)    while   Group   II!s   overall   response  indicated   they   were 
lacking  in  the   most    rudimentary   areas   such  as  basic   prob- 
ability and  statistics,    linear  programming,    calculus,    etc. 
Only  two   officers   in   Group  II   had  any  postgraduate   education, 
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These   disparities   in  relative  performance   of   Group   II    to 
Group   I    exist   despite  the  fact   that   the   mean   time  in 
billet  for   Group  II   is   19    months   -   only   two    months   less 
than  time  required  for   completion  of   the   Operations 
Analysis   Curriculum,    thus   indicating  that   on-the-job 
training   is   not   very   effective,,      From   these  facts   there 
exists   a   clear   implication  that   there  is   a   significant 
difference  in  the  technical  abilities   and  therefore   effective- 
ness  of   Group   I    and   Group   II   as   practicing   Operations 
Analysts.      The  least   one  could  say  is  that  Group  I   displays 
a  marked  advantage  over  Group  II   in  the  technical  tools 
needed  in  the  field  of   Operations  Analysis  „ 

One   might   say   this   is   not   surprising  -  any  logical 
person  would  safely  assume  an  Operations  Analysis  trained 
person  to   be   more   effective  than   one  not   so   trainedo       This 
conjecture  is   most   likely   true  in  highly   specialized  fields 
(such   as   microelectronics)  9    but   cannot   be   safely  assumed 
for   fields    such  as   Operations  Analysis  „      Operations  Analysis 
is    concerned   with   optimal   solutions   to   real  life  problems 
arrived  at  by  the  applications  of  all  sciences   and  the  rational 
logic   intrinsic   to   them.      Therefore  it   could  just  as   well  be 
said  that   an   officer   with  a   good   working  knowledge  of    "the 
sciences"  has   an   excellent    chance   of   being  an   effective 
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Operations  Analyst.      Although  all   examinees   in   Group   II 
have  Bachelor's    degrees   in  fields   requiring   the  basic    math- 
ematical and   scientific   technologies   used  by   the   Operations 
Analyst,    the   quantitative  results   of   this   study  indicate 
that   their   ability   to    effectively  apply   these  tools   to 
Operations  Analysis   problems   is   relatively  low. 

Why  is   all   this   important?      The   Navy  has   stated   that 
its   policy   in  the  future   will  be  to    continue  to    educate 

officers   at   the   Postgraduate  level   in   ever-increasing 

8 
numbers.        Whatever   their   reasons   may  be  for   arriving  at 

this   decision,    the   cost   of   training  these  officers   has   or 

will   be   expended  and   cannot  be  recovered.      It   therefore 

logically  follows   that   the   most    effective  use   of   these 

officers   will  yield   the   maximum   return  for   the  money   spent. 

In  the  past,    many   Operations  Analysis   trained   officers  have 

never  held   Operations  Analyst   billets   and   many   officers 

have  been  assigned   to   these  billets    so   long  after   their 

training  as   to   nullify   the   maximum    effectiveness   they   could 

9 
display.         It   is   highly   likely  that   this    situation   exists   in  other 

technical  fields    within  the   Navy0 

Q 

OPNAV    Instruction  1040.2    dated  9    December   1963?      2 

o 
Roman,    P.    Do,    Russel,    K.    B.,    and   Dunlop,    «J.    M. 

A   Suggested  Method  for  Measuring  the   Effectiveness   of   the 

Utilization  of    Technically   Trained   Personnel,    U.So    Naval 

Postgraduate   School,    Monterey,    California   1962+  s      9 
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These   results   clearly   indicate  that   the   Navy   should 
give    more   thought   to    the   problem    of   using   the   technical 
abilities    of   these   officers   more   effectively.       Specifically, 
a   review    of   present   career   planning   should  be   made  in 
hopes   of   arriving  at   a    method  of   duty  assignment   planning 
which   will  allow    technically  trained   officers   to   be  assigned 
associated  billets   compatible   with   their   education  at   the 
earliest   possible   date  after   completion  of   this    education. 
This   of   course   must  be  done  in  light   of   providing  the 
officer   with   the   operational   experience  necessary  for 
Military   Commands.  The  authors   are   certain  that   the 

Navy  is   aware   of   this   problem,    but    may   not   be   cognizant 
of   the  fact   that   the  use   of   officers   not   trained  in   such 
specialties   as   Operations  Analysis   results   in  a   large  loss 
of   effectiveness   that   could  be  realized  by   more  judicious 
use  of   the  available   corps   of   technically   trained   officers . 


10SECNAV   Instruction  1520.4   dated   7   March   1963s      2 
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Ill  o      Prediction   of   performance  in   Operations  Analysis 
Training. 

During   the  gathering  and  analysis   of   the   data  for   the 

previous   parts   of   this   paper   the  authors   discovered   what 

appeared   to   be  another   fruitful  avenue   of   investigation; 

the  prediction  of   performance  in  the  Operations  Analysis 

Curriculum.      Since   QoPcR.    is   accepted  universally  as   a 

measure   of   a   person's   knowledge  of   the  technical   tools 

acquired   during  his   training,    it   would  be   especially   welcome 

if    some   means    of   predicting,    before  training   commenced,    the 

approximate   skill  any  particular   person,    or    more  appropriately, 

any   group   of   persons    would  aquire.      If   such  a   predictor 

could  be   developed,    any  input   group   to   the  Operations 

Analysis    Curriculum    could  be   selected   so   as   to   yield   maximum 

benefit   to   the    Navy  for   the  time  and   money   expended   on   the 

training. 

In  an  attempt   to   develop   some   prediction  relationships, 

a   number   of    statistics   were   considered  as   variables.       Using 

QoPoR.    as   the  variable  to   be  predicted  nine   other   statistics 

were   treated   as   the  predictors; 

1.  Score   on   Part   I    of   the  test   vehicle. 

2.  Verbal   score  on   the  Graduate   Record    Examination. 

3.  Quantitative   score   on  the   Graduate   Record   Examination, 
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2f.      Advanced   score   on  the   Graduate   Record   Examination. 

5.  Time   since  graduation  or   time  in  an   Operations 
Anslysis   billet  . 

6.  Score   Part   II   of   the  test   vehicle^, 

7.  Score   Part   III   of   the   test   vehicle. 

8.  Category  as    defined  previously  for   responders. 

9.  Sum   of   Parts   I9    II 9    and  III. 

A   standard   linear   regression  analysis    (see  Appendix  IV) 
was   performed   on  a   CDC    160J+   to   obtain  the  prediction 
equations.       The   level   of    significance   on   the   F-test   used 
was   fixed  at    0.01   for  all   runs.      As   previously   mentioned 
in   Part  II    of   this   paper   one  run   was   made  using   the   scores 
of   Part  II   and  then   Part   III    of   the  test   vehicle  as   the 
dependent   variable.      Many   other   equations   were   also   obtained, 
not    with   the  purpose   of   predicting   Q.P.R,    specifically  but 
for   obtaining  as   much   information  about  the   correlations   of 
the  various   statistics   as   possible.      A   complete   summary   of 
these   equations   is   contained  in  Appendix  IV.      Initially   three 
different   groups   of   data   points   were   useds 

I .  All    Operations  Analysis    students   and   graduates 
for   which   Q»PoRo8    and   the   GoR0E0    scores    were  available. 

II.  The  class   of   1965° 

III.  The   class   of   1965  and   previous   Operations  Analysis 
graduates   whose   statistics    were   available. 
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From   each  group  of   data  points   eight  regression 
equations   were   obtained.      All  these   equations   are  tabulated 
in  Appendix  IV  „      The   equations   of   interest   in   predicting 
QoP.Ro    are  necessarily   those   containing  only  independent 
variables    -which   are  readily  available  before  a   person  or 
person's   start   the   course   of   instruction.        Of   the  group 
of   variables   used  in  the  analysis   only   Part   I    and   the   G.R.E 
scores   would  be   obtainable  prior   to    enrolement. 

The  following  is   the  regression  analysis   and   equations 
obtained  in  the  order   of   the  groups   of   data   points. 

GROUP   I 


Regression       Std.    Error   of 
Coefficient      Reg.    Coef. 


Var- 

iable 

Standard 

No. 

Mean 

Deviation 

2 

50.318 

6.349 

3 

579.649 

108.514 

4 

673.377 

86.551 

5 

308.701 

248.352 

1 

1.938 

.625 

.031 

.011 

.001 

.001 

.001 

.001 

.000 

.000 

Multiple   Correlation  Coefficient      «5978 


Computed 

Partial 

Variance 

Prop,,    Var. 

T  Value 

Corr.    Coef. 

Added 

Cum. 

2,778 

.311 

7.670 

.259 

1.466 

.170 

1.672 

,056 

1.639 

.189 

.699 

.024 

1.446 

.168 

.553 

.019 
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GROUP 

II 

Var- 

iable 

Standard 

Regression 

Std.    Error   of 

No. 

Mean 

Deviation 

Coefficient 

Reg.    Coef. 

2 

52,228 

5*365 

.032 

.013 

3 

610.833 

76.322 

-.001 

.001 

4 

675.556 

95o900 

.001 

.001 

5 

545.000 

91.853 

.001 

.001 

1 

2.043 

.410 

Multiple   Correlation  Coefficient      .6142 


Computed 
T   Value 


Partial 
Corr.    Coef. 


Variance 
Added 


Prop,    Var 
Cum. 


2.483 

-.993 

.662 

1.571 

c407 
-.176 
.118 
.272 

GROUP 

1.747 
.037 
.147 
.292 

III 

.296 
.006 
.025 
.050 

Var- 
iable 
No. 

M 

ean 

Standard 
Deviation 

Regression 
Coefficient 

Std. 
Reg. 

Error   of 
Coef. 

2 
3 

h 
5 
1 


52.433 
607.111 

682.889 
554.444 
2.103 


6,017 

82.754 

91.617 

93-628 

■  413 


022 
001 
001 
002 


.010 
.001 
.001 
.001 


Multiple   Correlation  Coefficient 


6118 


Computed 

Partial 

Variance 

Prop,    Var. 

T   Value 

Corr,    Cc 

ef . 

Added 

Cum. 

2.289 

.340 

1.652 

.220 

-  .943 

-.148 

.003 

.000 

.773 

.121 

.292 

.039 

2.712 

.394 

.864 

.115 
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It   is   readily   seen  from   the  above   equations   that   the 
multiple   correlation  coefficients   of   the  three  groups    of 
data  points  are  remarkably  constant    (0.5978,    0.6li+2, 
0.6118).       The  authors   found   this    somewhat   surprising 
since  previous   analysis   of   data  had  indicated  that   the 
Q.P.R.    of   the  1966   class    would  not  have   settled   down 
into   the  same  pattern  as   those  of   the  1965    class   and   the 
graduates.      It  had  further   been  hypothesized   that   the 
data   points   obtained  from   graduates    would  be   somewhat 
distorted   due  to   their   high    mean   Q.PoR.    and   the  fact 
that   since   they   were   not   in   school   they   would  have   lost 
some   of   their    "test   taking  ability."      Even  though   the 
multiple   correlation   coefficients   are  fairly   constant   over 
the  three  groups   of   data  it  is  apparent  that   the  class   of 
1966   does    degrade   the  results   and  it   is   therefore  felt   that 
either   the  second   or   third   equation   should  be  used  for 
prediction. 

As   time   passes   and   more   data   points  are   obtained  it 
is   felt   that   this    method   will  yield   ever  better   prediction 
equations.      The  authors   leave  it   to   the  readers   to   decide 
if   the  present    equations    with  approximately  0.34   as   the 
standard   error   of    estimate  is    sufficiently  accurate  for 
their   application.       The  authors   do   feel   that   the  results 
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obtained   thus   far   strongly   support   the  hypothesis   that   the 
predictors    (Part   I  „    G0RoE0    scores)    can  be  used  to    select 
the   makeup   of   a   class   or   predict   the  performance  of  a 
previously   selected  group „      It   is   further   felt   that   these 
predictors   and   methodology   could  be  applied  to   other 
curricula   with    equally  promising  results. 
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CHAPTER  III 
CONCLUSIONS  AND  RECOMMENDATIONS 

Reliability  and  Validity  determinations   for   the  test 
instrument  have   shown  that   the  basic   tool   of   the  proposed 
methodology  is   a  feasible  and   meaningful    model   with   which   to 
measure  technical  resources.      Admittedly  there   are  a  number 
of   necessarily   subjective    "proofs"   included  in  this   analysis, 
but   the  authors   feel   that   quantitative   determinations   have 
served  to   augment   these    "proofs,"   and  provide  a   creditable 
coalescence  leading  to  an  overall  high   degree  of  adequacy  for 
the  test   vehicle.      This   logical   trajectory  had  led  the  authors 
to   the  conclusion  that  this  test   does   measure  relative 
percentages   of   technical  resources   which  in  turn  gives  an 
excellent  indication   of    effectiveness   in  billets   requiring  these 
resources. 

Analysis   of   the  data  garnered  from   this   test    model   on 
the   Operations  Analysis    Curriculum  has   objectively  indicated 
that: 

(1)  The  percentage  of   technical   resources   retained 
decreases   logarithmically    (approximately)    with   time. 

( 2 )  On-the-job   training  in   Operations  Analyst   billets   is 
significantly  inferior   to   formal   Operations  Analysis   training  - 
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especially   in  the   area    of   learning   fundamental   technical 
knowledge « 

(3)    If   technical  ability  is   utilized  immediately  after 
it's   acquisition,    loss   of   this   resource   occurs   at   a   slower 
rate. 

(i+)    Although   Operations  Analysis   is   a   field  utilizing 
mainly   the  basic    sciences    (Mathematics,    Physics,    Probability, 
etc.),    College   Graduates    ( Non  Operations  Analysis   trained) 
holding   degrees    which   require  knowledge   of    most   of   these 
subjects   do   not   possess   nor   readily   learn  the  basic   tools 
of   the   Operations  Analyst. 

(5)  Graduates   of   the   Operations  Analysis    Curriculum 
(regardless   of   time   since   completion  of   course   or   use   or 
non-use   of   technical  abilities)    display   three    (3)    to   four    (4) 
times   the  technical  knowledge   of   College  Graduates    who   have 
held   Operations  Analyst   billets   for  an  average   of   19    months. 

(6)  Use   of  Part   I  of    the   test   instrument   and   the   three 
parts    of   the   graduate   achievement   tests    taken  by  all   students 
of   USNPGS   has   yielded  a   quantitative   method   of   augmenting 
the   screening   of   prospective   students   to   the   Operations 
Analysis    Curriculum. 

From   these   conclusions   it   is   recommended   that; 
(1)    Further   investigation  of   this    methodology  be   under- 
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taken.      It   is    obvious   that   the   sample   sizes   obtained   were 
not   large   enough  to    make    "bullet   proof"   statistical    conclusions 
but   rather   serve   as   indications   that   the   stated  postulates 
concerning   the  use  of   technical  abilities   and   the   effectiveness 
of   various   groups   of   officers  holding   Operations  Analyst 
billets   are   essentially   correct. 

(2)  The   significant   disparity  in  technical  abilities    of 

the  two   groups   of   officers   mentioned  in   5.    (above)    definitely 
suggests   that    Naval   Officer   assignment   policies   be  reviewed 
in  hopes   of   assigning  as    many   Operations  Analysis   trained 
officers   to    Operations  Analyst  billets   as   is   consistent   with 
other   requirements. 

(3)  Use   of   the  regression   equations   and   correlation 
coefficients   presented  in  Appendix  IV    would  be   of    consider- 
able use  for   Operations  Analysis    Curriculum   officials   in 
balancing   the  levels   of    Operations  Analysis   Classes   in  any 
manner   consistent    with   the   ever   changing  policies   and  needs 
of   the   Postgraduate   School. 

(2f)    If   future  investigations    of   this   type   are  undertaken 
a   more   effective   method   of   insuring   the  test   subjects   complete 
the  required   questionaire  and  test   instrument   should  be  invoked. 
The   authors    were  forced  to   appeal   to   the   examinees    sense 
of   responsibility,    their   response  being   completely  voluntary. 
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It   is   felt   that   in   order   to   obtain   meaningful   sample   sizes 
from    which   learning  and  forgetting   curves   can  be  accurately 
constructed,    and   measures   of   effectiveness   deduced, 
completion  of    the  test   vehicle  should,    in   some   manner,    be 
made   mandatory. 
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CATEGORY    1A         INPUT    OPERATIONS   ANALYSIS 

CLASS    1966    -    PART    ONE    TAKEN 
DURING    FIRST    TERM   IN 
CURRICULUM 


Examinee 

PART    I 

1 

50.6 

2 

52.6 

3 

35.6 

4 

45.6 

5 

54.6 

6 

50.6 

7 

53.6 

8 

45.6 

9 

33.6 

10 

47.6 

11 

51  .6 

12 

54.6 

13 

49.6 

14 

60.6 

15 

38.6 

16 

38.0 

17 

44  •  6 

IS 

47.6 

19 

43.6 

20 

46.6 

21 

53.6 

22 

51.6 

23 

37.6 

24 

43.6 

25 

50.6 

26 

i+6.6 

27 

58.6 

28 

44.0 

29 

52.6 

30 

50.6 

31 

48.6 

32 

49.6 

33 

54.6 

34 

41.6 

Mean   =  47.9 
Standard   Deviation  =      6.4I 
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RELIABILITY    CALCULATIONS 


Data   from   Table  #1 
Equations   from   Chapter   II 


n^-  Reliability  For  Test  Tnst 


RUM£NT. 


FL  =  U  -  K 


u. 


\r2      } 


P. 


T* 


R*  =  [i  -  nil  _  [v.  77 1 

L  /oo.^7£J       j_y/a.2,/J  ' 

R^   =0.90  ?4. 


u_  _    y.77 


N  te 


IE. 


o-o  ice  -c  0.-3  . 


Q.E.D 


hk 


'S    * 


APPENDIX    III 

Learning,    Relearning 
and   Forgetting   Curves 


k5 


KEY    TO    GRAPHS 

Weighted  Adjusted   Score  is    the   ratio   of   the  Mean  Score 
of  a  particular   category  to   the  individual's    score   on 
Part   I    multiplied  by   the  individual's   score   on   Part   II. 
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APPENDIX  IV 

REGRESSION  EQUATIONS 

AND 

CORRELATION  COEFFICIENTS 


50 


GROUP  I 
Regression  Equations  using  all  Operations  Analysis  students  and 
graduates  for  which  QPR,    and  the  G.R.E.    scores  were  available. 
Variable  No. 


1 

Quality  Point  Rating 

2 

Part  I  Score 

3 

G.R.E.  Verbal  Score 

4 

G.R.E.  Quantitative  Score 

5 

G.R.E.  Advanced  Score 

6 

Time  (in  billet  or  out  of  school) 

7 

Part  II  Score 

8 

Part  III  Score 

CORRELATION  COEFFICIENTS 

ROW   1 

1.00000    .42051    .39584    .37374    .50869 

ROW   2 

.42051   1.00000    .47756    .30214    .42939 

ROW   3 

.39584    .47756   1.00000    .19532    .41359 

ROW      4 

.37374         .30214        .19532      1.00000        .35157 

ROW   5 

.50869    .42939    .41359    .35157   1.00000 


SAMPLE  SIZE  77 

NO.  OF  VARIABLES  5     NO.  OF  VARIABLES  DELETED  3   (FOR  VARIABLES 

DEPENDENT  VARIABLE  IS  NO  NO.  1         DELETED,  SEE  BELOW) 

COEFFICIENT  OF  DETERMINATION    .3574 
MULTIPLE  CORR.  COEFFICIENT      .5978 


SUM  OF  SQUARES  ATTRIBUTABLE  TO  REGRESSION 
SUM  OF  SQUARES  OF  DEVIATION  FROM  REGRESSION 

VARIANCE  OF  ESTIMATE    .26456 
STD.  ERROR  OF  ESTIMATE  .51436 

INTERCEPT  (A  VALUE)   -1.14097 
STD.  ERROR  OF  INTERCEPT  .57577 
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GROUP  II 
Regression  Equations  using  just  the  class  of  1965< 


Variable  No, 

1 

Quality  Point  Rating 

2 

Part  I  Score 

3 

G.R.E.  Verbal  Score 

4 

G.R.E.  Quantitative  Score 

5 

G.R.E,  Advanced  Score 

6 

Time  (in  billet  out  of  school) 

7 

Part  II  Score 

8 

Part  III  Score 

CORRELATION  COEFFICIENTS 

ROW   1 

1.00000    .31319    .49274   .43350    .54445 

ROW   2 

.31319   1.00000    .43538   .26674    .09552 

ROW   3 

.49274    .43538   1.00000   .44631    .37425 

ROW   4 

.43350    .26674    .44631  1.00000    .45348 

ROW   5 

.54445    .09552    .37425   .45348   1.00000 


SAMPLE  SIZE  36 

NO.  OF  VARIABLES  5    NO.  OF  VARIABLES  DELETED  3  (FOR  VARIABLES 

DEPENDENT  VARIABLE  IS  NOW  NO.  1        DELETED ,  SEE  BELOW) 


COEFFICIENT  OF  DETERMINATION   .3772 
MULTIPLE  CORR.  COEFFICIENT     .6142 

SUM  OF  SQUARES  ATTRIBUTABLE  TO  REGRESSION     2.22289 
SUM  OF  SQUARES  OF  DEVIATION  FROM  REGRESSION   3.66991 

VARIANCE  OF  ESTIMATE     .11838 
STD.  ERROR  OF  ESTIMATE   .34407 

INTERCEPT  (A  VALUE)     -.10242 
STD.  ERROR  OF  INTERCEPT   .64905 
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GROUP  III 
Regression  Equations  using  the  class  of  1965  and  previous 
Operations  Analysis  graduates  whose  statistics  were  available. 
Variable  No. 


1 

Quality  Point  Rating 

2 

Part  I  Score 

3 

G.R.E.  Verbal  Score 

4 

G.R.E.  Quantitative  Score 

5 

G.R.E.  Advanced  Score 

6 

Sum  of  Parts  I,  II,  and  III 

7 

Part  II  Score 

8 

Part  III  Score 

CORRELATION  COEFFICIENTS 


ROW       1 


1.00000 

.34292 

.41106 

.36267 

.46906 

ROW       2 
.34292 

1.00000 

.47445 

.28329 

.14136 

ROW      3 
.41106 

.47445 

1.00000 

.41974 

.35087 

ROW      4 

.36267 

.28329 

.41974 

1.00000 

.51502 

ROW       5 
.46906 

.14136 

.35087 

.51502 

1.00000 

SAMPLE  SIZE  45 

NO.  OF  VARIABLES  5     NO.  OF  VARIABLES  DELETED  3  (FOR  VARIABLES 

DEPENDENT  VARIABLE  IS  NOW  NO.  1        DELETED,  SEE  BELOW) 

COEFFICIENT  OF  DETERMINATION    .3743 
MULTIPLE  CORR.  COEFFICIENT      .6118 

SUM  OF  SQUARES  ATTRIBUTABLE  TO  REGRESSION      2.81137 
SUM  OF  SQUARES  OF  DEVIATION  FROM  REGRESSION    4.69891 


VARIANCE  OF  ESTIMATE 
STD.  ERROR  OF  ESTIMATE 

INTERCEPT  (A  VALUE) 
STD.  ERROR  OF  INTERCEPT 


.11747 
.34274 

.02897 
.53596 
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GROUP  IV 
Regression  Equations  using  only  returns  from  graduates  of  the 
Operations  Analysis  Curriculum  or  officers  holding  Operations 
Analyst  billets . 
Variable  No. 


1 

2 
3 
4 
5 

Time  (in  billet  o: 
Part  I  Score 
Part  II  Score 
Part  III  Score 
Category  (2A=1.0S 

CORRELATION 

COEFFICIENTS 

ROW       1 
1.00000 

-.08659 

.20970 

.12543 

ROW       2 
-.08659 

1.00000 

■=.18069 

.36119 

ROW       3 
.20970 

-.18069 

1.00000 

-.72901 

ROW      4 
.12543 

.36119 

-. 72901 

1.00000 

SAMPLE  SIZE  34 
NO  OF  VARIABLES  4 


NO.  OF  VARIABLES  DELETED  1  (FOR  VARIABLES 


DELETED,  SEE  BELOW) 


DEPENDENT  VARIABLE  IS  NOW  NO.  3 

COEFFICIENT  OF  DETERMINATION    .6740 
MULTIPLE  CORR.  COEFFICIENT      .8210 


SUM  OF  SQUARES  ATTRIBUTABLE  TO  REGRESSION      2526.80694 
SUM  OF  SQUARES  OF  DEVIATION  FROM  REGRESSION    1222.22490 

VARIANCE  OF  ESTIMATE     40.74083 
STDo  ERROR  OF  ESTIMATE     6.38285 

INTERCEPT  (A  VALUE)        5.6046O 
STD.  ERROR  OF  INTERCEPT    9.01661 
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